Distributed Continuous Quality Assurance Process for Evaluating QoS of Performance-Intensive Software

Performance-intensive software is increasingly being used on heterogeneous combinations of OS, compiler, and hardware platforms. Examples include reusable middleware that forms the basis for scientific computing grids and distributed real-time and embedded systems. Since this software has stringent quality of service (QoS) requirements, it often provides a multitude of configuration options that can be tuned for specific application workloads and run-time environments. As performance-intensive systems evolve, developers typically conduct quality assurance (QA) tasks in-house, on a small number of configurations to identify and alleviate overhead that degrades performance. QA performed solely in-house is inadequate because of time/resource constraints, and insufficient to manage software variability, i.e., ensuring software quality on all supported target platforms across all desired configuration options. This paper addresses limitations with in-house QA for performance-intensive software by applying the Skoll distributed continuous quality assurance (DCQA) processes, to improve software performance iteratively, opportunistically, and efficiently around-the-clock in multiple, geographically distributed locations. It describes model driven tools that allow Skoll users to capture the system’s axes of variability (such as configuration options, QoS strategies, and platform dependencies). It describes experiments that apply Skoll to evaluate and improve the performance of DRE component middleware on a range of platforms and configuration options. The results show that automatic analysis of QA task results can significantly improve software quality by capturing the impact of software variability on performance and providing feedback to help developers optimize performance.